Papers with LLM-based evaluation method
IFIR: A Comprehensive Benchmark for Evaluating Instruction-Following in Expert-Domain Information Retrieval (2025.naacl-long)
Copied to clipboard
| Challenge: | Current information retrieval systems struggle to handle complex instructions, despite its critical importance . current models struggle to follow complex instructions in real-world applications, resulting in user-specific tasks. |
| Approach: | They propose a benchmark to evaluate instruction-following information retrieval in expert domains. |
| Outcome: | The proposed method improves on existing models and provides valuable insights to guide future advancements in retrieval. |
C3: A Bilingual Benchmark for Spoken Dialogue Models Exploring Challenges in Complex Conversations (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent developments in spoken dialogue models have created a gap in understanding their effectiveness in comprehending and emulating human conversations. |
| Approach: | They present a benchmark dataset which comprises 1,079 instances in English and Chinese to examine their effectiveness in emulating human conversations. |
| Outcome: | The proposed model outperforms existing models in English and Chinese by using an LLM-based evaluation method that closely aligns with human judgment. |